Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(ticdc): add batch field to resolveTs #5342

Merged
merged 2 commits into from
May 18, 2022
Merged

sink(ticdc): add batch field to resolveTs #5342

merged 2 commits into from
May 18, 2022

Conversation

CharlesCheung96
Copy link
Contributor

@CharlesCheung96 CharlesCheung96 commented May 6, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #5398

What is changed and how it works?

  1. Add batch field to resolveTs

  2. Added batch Resolved semantics support to Sink's FlushRowChangedEvents interface:

    • With resolved.Batch == false, TiCDC guarantees that all the Events whose commitTs is less than or equal to resolved.Ts are sent to Sink through EmitRowChangedEvents.
    • With resolved.Batch == true, TiCDC guarantees that all events whose commitTs is less than 'resolved.Ts' are sent to Sink.

Check List

Tests

  • Unit test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change

Side effects

  • Possible performance regression
  • Increased code complexity

Release note 

`None`.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 6, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • hi-rustin

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 6, 2022
@CharlesCheung96 CharlesCheung96 added the area/ticdc Issues or PRs related to TiCDC. label May 7, 2022
@CharlesCheung96 CharlesCheung96 changed the title Add batch resolve mechanism to support splitting large transactions sink(ticdc): Add batch resolve mechanism to support splitting large transactions May 7, 2022
@CharlesCheung96 CharlesCheung96 force-pushed the add_batch_reolve_mechanism branch from f6ae747 to d0f7a81 Compare May 11, 2022 17:36
@CharlesCheung96 CharlesCheung96 changed the title sink(ticdc): Add batch resolve mechanism to support splitting large transactions sink(ticdc): Add batch field to resolveTs May 11, 2022
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. labels May 11, 2022
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@CharlesCheung96 CharlesCheung96 changed the title sink(ticdc): Add batch field to resolveTs sink(ticdc): add batch field to resolveTs May 12, 2022
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@CharlesCheung96
Copy link
Contributor Author

/run-kafka-integration-test
/run-leak-test

2 similar comments
@CharlesCheung96
Copy link
Contributor Author

/run-kafka-integration-test
/run-leak-test

@CharlesCheung96
Copy link
Contributor Author

/run-kafka-integration-test
/run-leak-test

@CharlesCheung96
Copy link
Contributor Author

/run-leak-test

@CharlesCheung96
Copy link
Contributor Author

/run-check-issue-triage-complete

@CharlesCheung96 CharlesCheung96 marked this pull request as ready for review May 12, 2022 11:56
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 12, 2022
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@codecov-commenter
Copy link

codecov-commenter commented May 12, 2022 •
edited
Loading

Codecov Report

Merging #5342 (c507333) into master (687e248) will decrease coverage by 0.3650%.
The diff coverage is 52.1852%.

Flag Coverage Δ
cdc 60.6577% <52.1852%> (+0.0591%) ⬆️
dm 51.9519% <ø> (-0.5177%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #5342        +/-   ##
================================================
- Coverage   56.1240%   55.7589%   -0.3651%     
================================================
  Files           522        527         +5     
  Lines         65325      69605      +4280     
================================================
+ Hits          36663      38811      +2148     
- Misses        25094      27069      +1975     
- Partials       3568       3725       +157

@ti-chi-bot
Copy link
Member

@nongfushanquan: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@CharlesCheung96 CharlesCheung96 added the status/ptal Could you please take a look? label May 13, 2022
3AceShowHand
3AceShowHand reviewed May 13, 2022
View reviewed changes
cdc/model/mounter.go Outdated Show resolved Hide resolved
3AceShowHand
3AceShowHand reviewed May 13, 2022
View reviewed changes
cdc/model/mounter.go Outdated
// ResolvedTs is the resolved timestamp of sink module.
type ResolvedTs struct {
Ts uint64
Batch bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we can add some comments here, what is the usage of Batch, and why do we need it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is commented on the FlushRowChangedEvents of sink module.

@CharlesCheung96 CharlesCheung96 force-pushed the add_batch_reolve_mechanism branch 2 times, most recently from 2f4f58c to 0aae81c Compare May 16, 2022 10:00
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 16, 2022
@CharlesCheung96 CharlesCheung96 force-pushed the add_batch_reolve_mechanism branch from 0aae81c to c507333 Compare May 17, 2022 02:38
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 17, 2022
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

amyangfei
amyangfei reviewed May 17, 2022
View reviewed changes
cdc/entry/mounter.go Show resolved Hide resolved
cdc/processor/pipeline/sink.go Show resolved Hide resolved
@CharlesCheung96 CharlesCheung96 force-pushed the add_batch_reolve_mechanism branch from c507333 to 7fffb30 Compare May 17, 2022 05:50
@amyangfei
Copy link
Contributor

This PR doesn't change any logic, only adds the batch field, is it right?

@CharlesCheung96
Copy link
Contributor Author

This PR doesn't change any logic, only adds the batch field, is it right?

Yes

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 17, 2022
@CharlesCheung96 CharlesCheung96 force-pushed the add_batch_reolve_mechanism branch from 7fffb30 to 458705f Compare May 17, 2022 10:13
@CharlesCheung96 CharlesCheung96 force-pushed the add_batch_reolve_mechanism branch from 63f12d4 to 434508b Compare May 18, 2022 03:41
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

Rustin170506
Rustin170506 approved these changes May 18, 2022
View reviewed changes
Copy link
Member

@Rustin170506 Rustin170506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 18, 2022
@Rustin170506
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 434508b

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 18, 2022
@Rustin170506 Rustin170506 removed the status/ptal Could you please take a look? label May 18, 2022
@Rustin170506
Copy link
Member

/run-dm-integration-test

@ti-chi-bot ti-chi-bot merged commit 7649bda into pingcap:master May 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3AceShowHand 3AceShowHand 3AceShowHand left review comments

@nongfushanquan nongfushanquan nongfushanquan approved these changes

@Rustin170506 Rustin170506 Rustin170506 approved these changes

@amyangfei amyangfei amyangfei approved these changes

@sdojjy sdojjy Awaiting requested review from sdojjy

Assignees
No one assigned
Labels
area/ticdc Issues or PRs related to TiCDC. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add batch field to resolveTs
7 participants
@CharlesCheung96 @ti-chi-bot @codecov-commenter @amyangfei @Rustin170506 @3AceShowHand @nongfushanquan